Using Deep Neural Networks to Learn Syntactic Agreement

We consider the extent to which different deep neural network (DNN) configurations can learn syntactic relations, by taking up Linzen et al.’s (2016) work on subject-verb agreement with LSTM RNNs. We test their methods on a much larger corpus than they used (a ⇠24 million example part of the WaCky corpus, instead of their ⇠1.35 million example corpus, both drawn from Wikipedia). We experiment with several different DNN architectures (LSTM RNNs, GRUs, and CNNs), and alternative parameter settings for these systems (vocabulary size, training to test ratio, number of layers, memory size, drop out rate, and lexical embedding dimension size). We also try out our own unsupervised DNN language model. Our results are broadly compatible with those that Linzen et al. report. However, we discovered some interesting, and in some cases, surprising features of DNNs and language models in their performance of the agreement learning task. In particular, we found that DNNs require large vocabularies to form substantive lexical embeddings in order to learn structural patterns. This finding has interesting consequences for our understanding of the way in which DNNs represent syntactic information. It suggests that DNNs learn syntactic patterns more efficiently through rich lexical embeddings, with semantic as well as syntactic cues, than from training on lexically impoverished strings that highlight structural patterns

Visual coherence of moving and stationary image changes

AbstractDetection thresholds were compared for moving and stationary oscillations with equivalent contrast changes. Motion was more detectable than stationary oscillation, and the difference increased with size of the feature (a Gaussian blob). Phase discriminations between a center and two flanking features were much better for motion than for stationary oscillation. Motion phase discriminations were similar to motion detection and were robust over increases in spatial separation and temporal frequency, but not so for stationary oscillations. Separate visual motion signals were positively correlated, but visual signals for stationary oscillation were negatively correlated. Evidently, motion produces visually coherent changes in image structure, but stationary contrast oscillation does not

Bayesian Inference Semantics: A Modelling System and A Test Suite

We present BIS, a Bayesian Inference Seman- tics, for probabilistic reasoning in natural lan- guage. The current system is based on the framework of Bernardy et al. (2018), but de- parts from it in important respects. BIS makes use of Bayesian learning for inferring a hy- pothesis from premises. This involves estimat- ing the probability of the hypothesis, given the data supplied by the premises of an argument. It uses a syntactic parser to generate typed syn- tactic structures that serve as input to a model generation system. Sentences are interpreted compositionally to probabilistic programs, and the corresponding truth values are estimated using sampling methods. BIS successfully deals with various probabilistic semantic phe- nomena, including frequency adverbs, gener- alised quantifiers, generics, and vague predi- cates. It performs well on a number of interest- ing probabilistic reasoning tasks. It also sus- tains most classically valid inferences (instan- tiation, de Morgan’s laws, etc.). To test BIS we have built an experimental test suite with examples of a range of probabilistic and clas- sical inference patterns

A Neural Model for Compositional Word Embeddings and Sentence Processing

We propose a new neural model for word embeddings, which uses Unitary Matrices as the primary device for encoding lexical information. It uses simple matrix multiplication to derive matrices for large units, yielding a sentence processing model that is strictly compositional, does not lose information over time steps, and is transparent, in the sense that word embed- dings can be analysed regardless of context. This model does not employ activation functions, and so the network is fully accessible to analysis by the methods of linear algebra at each point in its operation on an input sequence. We test it in two NLP agreement tasks and obtain rule like perfect accuracy, with greater stability than current state-of-the-art systems. Our proposed model goes some way towards offer- ing a class of computationally powerful deep learning systems that can be fully understood and compared to human cognitive processes for natural language learning and representation

GPR35 as a Novel Therapeutic Target

G protein-coupled receptors (GPCRs) remain the best studied class of cell surface receptors and the most tractable family of proteins for novel small molecule drug discovery. Despite this, a considerable number of GPCRs remain poorly characterized and in a significant number of cases, endogenous ligand(s) that activate them remain undefined or are of questionable physiological relevance. GPR35 was initially discovered over a decade ago but has remained an “orphan” receptor. Recent publications have highlighted novel ligands, both endogenously produced and synthetic, which demonstrate significant potency at this receptor. Furthermore, evidence is accumulating which highlights potential roles for GPR35 in disease and therefore, efforts to characterize GPR35 more fully and develop it as a novel therapeutic target in conditions that range from diabetes and hypertension to asthma are increasing. Recently identified ligands have shown marked species selective properties, indicating major challenges for future drug development. As we begin to understand these issues, the continuing efforts to identify novel agonist and antagonist ligands for GPR35 will help to decipher its true physiological relevance; translating multiple assay systems in vitro, to animal disease systems in vivo and finally to man

Assessing the Unitary RNN as an End-to-End Compositional Model of Syntax

We show that both an LSTM and a unitary-evolution recurrent neural network (URN) can achieve encouraging accuracy on two types of syntactic patterns: context-free long distance agreement, and mildly context-sensitive cross serial dependencies. This work extends recent experiments on deeply nested context-free long distance dependencies, with similar results. URNs differ from LSTMs in that they avoid non-linear activation functions, and they apply matrix multiplication to word embeddings encoded as unitary matrices. This permits them to retain all information in the processing of an input string over arbitrary distances. It also causes them to satisfy strict compositionality. URNs constitute a significant advance in the search for explainable models in deep learning applied to NLP

Assessing the efficacy and cost of detergents used in a primary care automated washer disinfector

Background: Cleaning of re-usable medical devices is a critical control point in the decontamination cycle, although defined end-points of the process are controversial. Objective: Investigate cleaning efficacy and cost of different detergent classes in an automated washer disinfector (AWD) designed for dental practice. Methods: Loads comprised test soiled dental hand instruments in cassettes and extraction forceps. Residual protein assayed using the International standard method (ISO 15883-5:2005) 1% SDS elution with ortho-phthalaldehyde (OPA) or GBox technology (on instrument OPA analysis). Short (60 minutes) and long (97 minutes) AWD cycles were used with four different classes of detergents, tap water and reverse osmosis water. Results: SDS elution analysis (N = 612 instruments) demonstrated four detergents with both wash cycles achieved equivalent cleanliness levels and below a threshold of 200 μg protein/instrument. GBox methodology (N = 575) using UK Department of Health threshold of 5 μg/instrument side demonstrated that tap water performed with the greatest efficacy for all types of instruments and cycle types. Conclusions: Using International standard methodology, different detergent classes had equivalence in cleaning efficacy. Cheaper detergents used in this study performed with similar efficacy to more expensive solutions. Findings emphasise the importance of validating the detergent (type and concentration) for each AWD

Potential prognostic marker ubiquitin carboxyl-terminal hydrolase-L1 does not predict patient survival in non-small cell lung carcinoma

<p>Abstract</p> <p>Background</p> <p>Ubiquitin Carboxyl-Terminal Hydrolase-L1 (UCH-L1) is a deubiquitinating enzyme that is highly expressed throughout the central and peripheral nervous system and in cells of the diffuse neuroendocrine system. Aberrant function of UCH-L1 has been associated with neurological disorders such as Parkinson's disease and Alzheimer's disease. Moreover, UCH-L1 exhibits a variable expression pattern in cancer, acting either as a tumour suppressor or promoter, depending on the type of cancer. In non-small cell lung carcinoma primary tumour samples, UCH-L1 is highly expressed and is associated with an advanced tumour stage. This suggests UCH-L1 may be involved in oncogenic transformation and tumour invasion in NSCLC. However, the functional significance of UCH-L1 in the progression of NSCLC is unclear. The aim of this study was to investigate the role of UCH-L1 using NSCLC cell line models and to determine if it is clinically relevant as a prognostic marker for advanced stage disease.</p> <p>Methods</p> <p>UCH-L1 expression in NSCLC cell lines H838 and H157 was modulated by siRNA-knockdown, and the phenotypic changes were assessed by flow cytometry, haematoxylin & eosin (H&E) staining and poly (ADP-ribose) polymerase (PARP) cleavage. Metastatic potential was measured by the presence of phosphorylated myosin light chain (MLC2). Tumour microarrays were examined immunohistochemically for UCH-L1 expression. Kaplan-Meier curves were generated using UCH-L1 expression levels and patient survival data extracted from Gene Expression Omnibus data files.</p> <p>Results</p> <p>Expression of UCH-L1 was decreased by siRNA in both cell lines, resulting in increased cell death in H838 adenocarcinoma cells but not in the H157 squamous cell line. However, metastatic potential was reduced in H157 cells. Immunohistochemical staining of UCH-L1 in patient tumours confirmed it was preferentially expressed in squamous cell carcinoma rather than adenocarcinoma. However the Kaplan-Meier curves generated showed no correlation between UCH-L1 expression levels and patient outcome.</p> <p>Conclusions</p> <p>Although UCH-L1 appears to be involved in carcinogenic processes in NSCLC cell lines, the absence of correlation with patient survival indicates that caution is required in the use of UCH-L1 as a potential prognostic marker for advanced stage and metastasis in lung carcinoma.</p